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© A computer, responsive to user input, controls 
the presentation of an audio-visual work to a user. 
Through the selection of several interactive learning 
options, support and reinforcement of the learning 
process is provided. In particular, the computer inter- 
acts with the user to challenge the user's under- 
standing of the audio-visual work. In connection with 
the presentation of foreign language works, the user 
interacts with the computer in role playing, transcrip- 
tion, translation, fill-in-the-blanks and speech repeti- 
tion activities designed to teach the user to speak 
the foreign language. Furthermore, as the audio- 
visual work is presented, a computer generated tran- 
scription or translation is displayed for user con- 
templation, and the user may interact with the com- 
puter by requesting the display of grammatical, cul- 
tural and geographic notes the further assist in the 
learning process. 
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TECHNICAL FIELD 

The present invention relates to teaching meth- 
ods, teaching aids and teaching devices and, in 
particular, to an interactive audio-visual system for 
teaching students a foreign language skill. 

BACKGROUND OF THE INVENTION 

Foreign languages are most often taught to 
students in classroom situations wherein interaction 
with the teacher, and sometimes other students, in 
conjunction with the use of study materials such as 
text books and audio tapes, is the preferred meth- 
od of instruction. Such classroom instruction ade- 
quately prepares students to handle routine com- 
munications tasks in the foreign country, such as 
ordering meals or asking for directions, but less 
than adequately prepares the student to engage in 
and understand foreign language conversations. 

One reason that the traditional instruction meth- 
ods are deficient is that the classes are generally 
directed to the instruction of the language in a 
formal or classic manner. Thus, the student re- 
ceives little or no training regarding the colloquial- 
ism, slang, grammar and syntax of the language as 
it is commonly used in conversations in the foreign 
country. Another reason that the traditional meth- 
ods are deficient is that the student is often in- 
structed by a teacher whose native language is not 
the foreign language being learned by the student. 
Thus, the student fails to master the proper accent 
or pronunciation required to comfortably commu- 
nicate in the foreign country. 

Another significant concern of foreign language 
students is that their conversation skills decay from 
non-use. Polishing of their previously learned skills 
often requires enrollment in a formal refresher 
course, or a trip to the foreign country coupled with 
an extended stay. This can be quite an expensive 
proposition, especially in view of the fact that 
courses or trips as often as once a year may be 
required. 

Accordingly, there is a need for a teaching 
method and system that prepares the foreign lan- 
guage student to speak in a conversational manner. 
Such a method or system should further facilitate 
the polishing of already acquired foreign language 
skills for those who do not have the opportunity to 
converse in the foreign language on a routine ba- 
sis. 

SUMMARY OF THE INVENTION 

The present invention comprises an interactive 
audio-visual system preferably utilized for teaching 
students certain skills, such as a foreign language. 
The system of the present invention includes a 



computer interconnected with an audio-visual 
playback device. Operation of the audio-visual 
playback device is controlled by a resident system 
operation program executed by the computer in 

5 response to student inputs. The system functions 
with the computer both controlling the playback of 
an audio-visual work by means of the audio-visual 
playback device and interacting with the student 
concerning the displayed audio-visual work. Such 

io interactions facilitate the process of learning the 
designated skill. 

The student may selectively choose to view the 
entire work from beginning to end, or alternatively 
view individually selected scenes. During viewing, 

15 the computer interacts with the user to challenge 
the user's understanding of both the audio-visual 
work and the skill being taught. The nature of the 
interaction between the user and the computer 
varies according to the skill being learned. Upon 

20 request, the system will display a synopsis of the 
entire work or a selected scene, overlay the text of 
the script (including a translation) on the displayed 
video images, provide definitions of user selected 
words and phrases and display notes on grammar. 

25 Furthermore, the system tests student comprehen- 
sion of the audio-visual work and the displayed skill 
by asking the student questions. 

The present invention preferably uses foreign 
language audio-visual works in the instruction and 

30 practice of a foreign language. Such works provide 
a culturally authentic source of foreign language 
material for teaching students the colloquialisms, 
slang, grammar, syntax and accent that are needed 
to prepare the student to conversationally interact 

35 in the foreign country or polish already acquired 
language skills. In connection with the presentation 
of such foreign language works, the user interacts 
with the computer in role playing, transcription, 
translation, fill-in-the-blanks, language comprehen- 

40 sion and speech repetition activities for the pur- 
pose of enhancing pronunciation and oral and read- 
ing comprehension skills. Furthermore, as the 
audio-visual work is presented, a computer gen- 
erated transcription or translation is displayed for 

45 user contemplation, and the user may interact with 
the computer by requesting the display of gram- 
matical, cultural and geographic notes that further 
assist in the learning process. 

50 BRIEF DESCRIPTION OF THE DRAWINGS 

A more complete understanding of the system 
of the present invention may be had by reference 
to the following Detailed Description when taken in 
55 conjunction with the accompanying Drawings 
wherein: 

FIGURE 1 is a block diagram of the interactive 
learning system of the present invention; 
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FIGURE 2 is a flow diagram for the functional 
operations and screen displays of the interactive 
learning system of the present invention when 
used as an interactive audio-visual foreign lan- 
guage skills maintenance tool; 
FIGURES 3 through 6 show representations of 
several menu screens displayed for the user to 
facilitate interactive user learning; and 
FIGURES 7 and 8 show representations of other 
screens displayed for the user to facilitate inter- 
active user learning. 

DETAILED DESCRIPTION OF THE DRAWINGS 

Reference is now made to FIGURE 1 wherein 
there is shown a block diagram of the interactive 
learning system 10 of the present invention. The 
system 10 includes a personal computer 12, an 
audio-visual playback device 14, a textual input 
device 16, a graphical input device 18, a video 
display 20 and an audio system 22. The means for 
interconnecting the computer 12 to the various 
components shown in FIGURE 1 is well known in 
the art. In the present embodiment, the video dis- 
play 20 is connected to the personal computer 12 
via a video cable 40. Furthermore, the personal 
computer 12 is connected to the audio-visual 
playback device 14, textual input device 16, graphi- 
cal input device 18 and audio system 22 via a bus 
42 which comprises both analog (including video 
cable) and digital communications lines. 

The personal computer 12 preferably com- 
prises an IBM AT-type (or equivalent) personal 
computer having a 386 (or better) processor 24, 2 
MB random access memory (RAM) 26, 20 MB 
mass storage (hard drive) 28, a disc drive 30 and a 
motion video overlay card 34. It will, of course, be 
understood that any computing system having 
equivalent functional and operational capabilities 
may be substituted for the personal computer 12. 
The disc drive 30 allows the computer 12 to re- 
ceive and access floppy data discs 32 containing 
program data and user data. Such data is accessed 
directly from the floppy disc 32 and/or stored by 
the computer in, and accessed from the RAM 26 
and/or mass storage (hard drive) 28 as necessary. 
The motion video overlay card 34 allows the com- 
puter 12 to control not only the display of video 
images output from the audio-visual playback de- 
vice 14 for display on the video display 20, but 
also to control the positioning and sizing of the 
images along with the overlay of computer gen- 
erated text and images if desired. 

The audio-visual playback device 14 preferably 
comprises a computer controllable video disc play- 
er 14a for receiving a video disc 36 and retrieving 
therefrom stored audio-visual data. The audio-vi- 
sual data stored on the video disc 36 preferably 



comprises an audio-visual work such as a motion 
picture, a television show or other broadcast. In the 
alternative, or in addition, the audio-visual playback 
device 14 may comprise a controllable CD-ROM 

5 player 14b for receiving and retrieving audio-visual 
data from a compact disc 38. 

The video display 20 preferably comprises a 
SVGA monitor for providing the display of high 
quality video images. The audio system 22 com- 

/o prises self-amplifying speakers or headphones with 
volume control to facilitate user reception of the 
audio portions of audio-visual works output from 
the audio-visual playback device 14 and displayed 
on the video display 20. The audio system 22 

75 further includes an audio input device, such as a 
microphone, to allow the user to input speech or 
other sounds into the computer 12 for subsequent 
processing or replay. 

The textual input device 16 comprises a stan- 

20 dard QWERTY-type alphanumeric keyboard, with 
function keys, through which the system user may 
both input alphanumeric text for processing and/or 
display by the computer 12 and select certain 
functions and operations for computer execution. 

25 The graphical input device 18 comprises either a 
joystick, mouse or trackball, as desired, to provide 
the computer user with a pointing and selecting 
device for on-screen selection of options and fea- 
tures in a well known manner (commonly referred 

30 to in the art as a "point-and-click" operation). It will, 
of course, be understood that other types of input 
devices may be used. For example, the video 
display 20 may further include a touch screen 
capability. In addition, the audio system 22 may be 

as used for audible input with the addition of currently 
available or subsequently developed speech rec- 
ognition functionality to the processor 24. 

The processor 24 controls operation of the 
system 10, including controlling all communications 

40 by and between the personal computer 12 and the 
audio-visual playback device 14, textual input de- 
vice 16, graphical input device 18, video display 20 
and audio system 22. The personal computer 12 
operates according to a system operation program 

45 downloaded from either the floppy disc 32 or the 
mass storage (hard drive) 28 to the RAM 26 for 
computer execution. The user of the system 10 
commands the functioning and execution of the 
system operation program, and thus the operation 

so of the system 10, by providing commands input via 
the textual and graphical inputs 16 and 18, respec- 
tively. 

The operation of the system 10 of the present 
invention will be described in connection with its 
55 preferred embodiment as an interactive audio-vi- 
sual foreign language skills maintenance tool. It 
will, of course, be understood that some, if not all, 
of the disclosed functionality provided by the sys- 
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tern 10 of the present invention may be equally 
applicable to and appropriate for other educational 
and skill training exercises. For example, the sys- 
tem 10 of the present invention may be used to 
teach not only verbal and reading comprehension 
skills with respect to a foreign language, but also 
other language skills and/or physical and technical 
skills, if desired. 

Regardless of the skill being taught, the user 
must first insert the appropriate video disc 36 for 
that skill into the video disc player 14a (or compact 
disc 38 into the CD-ROM player 14b). The video 
disc 36 (or compact disc 38) provides a storage 
media for the audio-visual and digital data used as 
source material to teach the selected skill to the 
user. Other data to be described associated with 
the subject matter recorded on the selected video 
disc 36, and necessary for instructing on the se- 
lected skill, is stored on a floppy disc 32. The 
digital data on the floppy disc 32 is accessed by 
the computer 12 using disc drive 30 in accordance 
with the executed system operation program in a 
manner to be described. 

In the preferred use of the system 10 as an 
interactive foreign language teaching tool, the video 
disc 36 provides a storage medium for a recorded 
feature length foreign language film. Other audio- 
visual works providing authentic foreign language 
source material (such as television shows, speech- 
es, musical and theatrical performances or sporting 
events) may also be recorded on video disc 36 for 
use in the system 10 of the present invention. The 
floppy disc 32 stores the digital data associated 
with the audio-visual data (for example, dialog tran- 
scription and translations, comprehension ques- 
tions, etc.). This digital data is specially prepared to 
accompany the selected audio-visual work record- 
ed on the video disc 36. The data on the floppy 
disc 32 thus provides additional information con- 
cerning the selected skill necessary for providing a 
more effective learning experience. 

A more complete understanding of the opera- 
tion of the system 10 of the present invention may 
be had by reference to FIGURE 2, along with 
reference to FIGURE 1 when appropriate, wherein 
FIGURE 2 illustrates a flow diagram for the func- 
tional operations performed by system operation 
program and the screens displayed when the sys- 
tem 10 is used as an interactive audio-visual for- 
eign language skills maintenance tool. It will, of 
course, be understood that some, if not all, of the 
steps and screen displays of the flow diagram of 
FIGURE 2 may be equally applicable to the in- 
struction of other skills using the system 10. Fur- 
thermore, any of the steps or screens may be 
altered or combined to display text, graphics or 
video data in alternate formats or to accommodate 
different subject matter and learning styles. 



Following the loading of the video disc 36 for a 
selected audio-visual work and the loading of the 
associated floppy disc 32 providing additional 
teaching material for the selected work, the system 

5 operation program is executed and the computer 
12 retrieves title data concerning the selected work 
from floppy disc 32 using disc drive 30. Such title 
data typically comprises the title of the audio-visual 
work, an identification of the makers (for example, 

10 producer and/or director) of the audio-visual work, 
copyright information for the audiovisual work, mak- 
ers of and copyright information for the system 
operation program (along with copyright information 
for the data on the floppy disc 32), and copyright 

75 information for any other reference materials (such 
as a dictionary or map) accessed by the computer 
12 during execution of the system operation pro- 
gram. The retrieved title data is displayed for the 
user on video display 20 using a title screen 40. 

20 Following display of the title screen 40, and 

further in accordance with the system operation 
program executed by the computer 12, a new user 
routing screen 42 is displayed on video display 20. 
The new user routing screen 42 asks the user to 

25 input whether the user has previously operated the 
system 10 and is familiar with the functions and 
features of the system. If the user replies nega- 
tively (see path 43), the system operation program 
skips directly to a help function (to be described in 

30 more detail below) that provides the user with the 
requisite information for efficiently and effectively 
using the system 10. If the user has used the 
system 10 before, and so answers (see path 41) in 
response to the display of the new user routing 

35 screen 42, the computer 12 will proceed with the 
next step in the execution of the system operation 
program. 

At this point, execution of the system operation 
program causes the main menu 44 (also shown in 

40 FIGURE 3) to be displayed for the user via video 
display 20. With display of the main menu 44, the 
title of the audio-visual work selected by the user is 
retrieved from floppy disc 32 and displayed at 46. 
The user is further given the choice of selecting 

45 from five options: "How to Use this System" 48; 
"About this Title (or Program)" 50; "Scene Menu" 
52; "View the Audio-Visual Work" 54; and "Quit" 
56. Selection of one of the five options is made 
using the graphical input 18 to position cursor 58 

so over the option and depress a select key in accor- 
dance with the well known computer point-and-click 
operation. In the alternative, with speech recogni- 
tion functionalities added to the processor 24, the 
user may audibly input the selection via the audio 

55 system 22. 

In response to either the entry of a negative 
response following display of the new user routing 
screen 42 (see path 43) or the selection of the 
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"How to Use this System" option 48 from the main 
menu 44 with either the cursor 58 and graphical 
input 18 (point-and-click) or audio system 22 input, 
further execution of the system operation program 
causes the computer 12 to display the How to Use 
this System Screen 60 (also shown in FIGURE 4). 
The How to Use this System Screen 60 includes a 
window 62, a window 64 and a return option 66. In 
window 62, the computer 12 retrieves and displays 
a list of the available system features, functions, 
options and operations 63 for the user to consider. 
The user may scroll through the list displayed in 
window 62 by inputting appropriate commands 
through either the textual or graphical inputs 16 
and 18, respectively, or audio system 22 input. 
Selection of one of the displayed topics in window 
62 with the cursor 58 and graphical input 18 (point- 
and-click) or audio system 22 input causes the 
computer 12 to retrieve and display a detailed 
explanation of the selected system feature, func- 
tion, option or operation 63 in window 64. The 
descriptions provided in windows 62 and 64 further 
include advice 63 on when to use or access a 
particular system feature, function, option or opera- 
tion. Such advice 63 assists the user in efficiently 
using the system 10 to provide for a more com- 
plete and effective skill learning experience. Fur- 
thermore, in response to the selection of the return 
option 66 with the cursor 58 and graphical input 18 
(point-and-click) or audio system 22 input, the sys- 
tem operation program executed by the computer 
12 will return the user to the main menu 44. 

In response to the selection of the "About this 
Title" option 50 from the main menu 44 with the 
cursor 58 and graphical input 18 (point-and-click) 
or audio system 22 input, further execution of the 
operational program causes the computer 12 to 
display the About this Title Screen 68 (also shown 
in FIGURE 5) including a window 77. The user is 
further given the choice of selecting from several 
options including: "About this Audio-Visual Work" 
70; "Ordering Information" 72; "Program Authors" 
74; and "Return" 76. In response to the selection 
of the "About this Audio-Visual Work" option 70 
with the cursor 58 and graphical input 18 (point- 
and-click) or audio system 22 input, the computer 
12 retrieves a summary of the work as a whole 
from the floppy disc 32 and displays the summary 
in the window 77 for user consideration. In re- 
sponse to the selection of the "Ordering Informa- 
tion" option 72 with the cursor 58 and graphical 
input 18 (point-and-click) or audio system 22 input, 
the computer 12 retrieves ordering information on 
the current title and other titles related to the cur- 
rent audio-visual work from the floppy disc 32 and 
displays the information in the window 77. In re- 
sponse to the selection of the "Program Authors" 
option 74 with the cursor 58 and graphical input 18 



(point-and-click) or audio system 22 input, the 
computer 12 retrieves biographical information con- 
cerning the maker(s) of the audio-visual work from 
the floppy disc 32 and displays the information in 

5 the window 77. Furthermore, in response to the 
selection of the "Return" option 76 with the cursor 
58 and graphical input 18 (point-and-click) or audio 
system 22 input, the operational program will return 
the user to the main menu 44. 

w In response to the selection of the "Scene 

Menu" option 52 from the main menu 44 with the 
graphical input 18 (point-and-click) or audio system 
22 input, further execution of the system operation 
program causes the computer 12 to display the 

75 Scene menu screen 78 (also shown in FIGURE 6) 
including a window 90 and a window 92. The user 
is further given the choice of selecting from five 
options: "Language" 80; "View Scene" 82; "Self- 
Check" 84; "Help" .86; and "Return" 88. In re- 

20 sponse to the selection of the "Language" option 
80 with the cursor 58 and graphical input 18 (point- 
and-click) or audio system 22 input, the computer 
12 displays in window 90 a list of available lan- 
guage choices for the display of written information 

25 by the computer in connection with the presenta- 
tion of the work. The user may then select either a 
native or familiar or other language for any future 
computer textual output. 

Furthermore, with respect to the display of the 

30 Scene menu screen 78, the computer 12 retrieves 
a list 91 of the scenes in the selected audio-visual 
work from the floppy disc 32, and displays the 
scene list 91 in window 92. The user may scroll 
through the scene list 91 displayed in the window 

35 92 by inputting appropriate commands through ei- 
ther the textual or graphical inputs 16 and 18, 
respectively, or audio system 22 input. Selection of 
one of the listed scenes with the cursor 58 and 
graphical input 18 (point-and-click) or audio system 

40 22 input causes the computer 12 to retrieve from 
the floppy disc 32 and display for the user in 
window 90 a synopsis 93 of the selected scene. In 
connection with the Scene menu screen 78, the 
user may further obtain, through entry of an appro- 

45 priate command, character information 95 compris- 
ing a list of characters along with a picture of each 
character and a brief character description. 

In response to the selection of the "View 
Scene" option 82 from the Scene menu screen 78 

so with the cursor 58 and graphical input 18 (point- 
and-click) or audio system 22 input, the video 
display 20 is cleared by the computer 12 and a 
signal is transmitted to the audio-visual playback 
device 14 to retrieve and playback the previously 

55 selected scene from the video disc 36. The se- 
lected scene is then displayed to the user for full 
screen viewing on the video display 20. During 
such full screen display, several interactive learning 
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features, to be described herein with respect to the 
description of the "View the Audio-Visual Work" 
option 54, are also made available (as generally 
indicated at reference 83) to the user for selection. 

In response to a user selection of the "Self- 
Check" option 84 from the Scene menu screen 78 
with the cursor 58 and graphical input 18 (point- 
and-click) or audio system input, further execution 
of the system operation program causes the com- 
puter 12 to display the Self-Check screen 96 
shown in FIGURE 7 on the video display 20. The 
scene selected by the user is displayed by the 
computer 12 for user viewing in window 98. When 
display of the scene is completed, the computer 
retrieves a series of questions and answers 97 
regarding the selected scene from the floppy disc 
32 and displays the questions in window 100 for 
user consideration. Preferably, such questions are 
designed to test user oral and/or reading com- 
prehension of the selected scene. Through use of 
the included "Language" option 102, the user may 
select the language with which the questions (and 
answers) are displayed. Answers to questions may 
either be formulated in the head of the user or 
input into the computer 12 via the textual input 16 
or audio system 22 input. When requested by the 
user by selecting window 104 with the cursor 58 
and graphical input 18 (point-and-click) or audio 
system 22 input, the computer 12 will retrieve from 
floppy disc 32 and display answers to each of the 
questions in window 104. Audibly input answers 
are processed in accordance with the speech rec- 
ognition functionality of the processor 24 and com- 
pared to the correct answers. Following either com- 
puter 12 checking of input answers for correctness, 
or user review of the formulated answers for cor- 
rectness, a selection of either the "Play Video" 
option 106 (to continue with the playback of the 
audio-visual work), or the "Scene List" option 108 
(to return to the Scene menu screen 78) may be 
made. 

In response to the selection of the "Help" 
option 86 from the Scene menu screen 78 with the 
cursor 58 and graphical input 18 (point-and-click) 
or audio system 22 input, the system operation 
program executed by the computer 12 returns to 
the How to Use This System menu screen 60 
described above. Unlike the exit from the How to 
Use This System menu screen 60 back to the Main 
menu 44 described above, when the user enters 
the How to Use This System menu from the Scene 
menu screen 78, selection of the return option 66 
will bring the user back to the Scene menu screen. 

Furthermore, in response to the selection of the 
"Return" option 88 from the Scene menu 78 with 
the cursor 58 and graphical input 18 (point-and- 
click) or audio 22 input, the system operation pro- 
gram will return the user to the main menu 44. 



In response to the selection of the "View the 
Audio-Visual Work" option 54 from the main menu 
44 with the cursor 58 and graphical input 18 (point- 
and-click) or audio system 22 input, the computer 

5 12 will transmit a control signal over bus 42 to the 
video-disc player 14a commanding the initiation of 
the playback from the beginning of the audio-visual 
work stored on the loaded video disc 36. The 
audio-visual work is displayed for the user, using a 

70 full screen, on the video display 20. Furthermore, 
the user may select through the input of appro- 
priate commands via either the textual or graphical 
inputs 16 and 18, respectively, to access the inter- 
active features 125 of the system 10 of the present 

75 invention in conjunction with the viewing of the 
audio-visual work. With some of these selections, 
the screen 110 of FIGURE 8 replaces the full 
screen display. 

The selected audio-visual work is displayed for 

20 user viewing in window 112 of screen 110. The 
user is further given the choice of selecting from 
six options: "Replay" 114; "Stop-Continue" 116; 
"Notes" 118; queuing controls 120; "Scene Menu" 
122; and "Return" 124. In response to the selection 

25 of the "Replay" option 114 with the cursor 58 and 
graphical input 18 (point-and-click) or audio system 
22 input, the computer 12 will direct the audio- 
visual playback device 14 to replay the currently 
displayed scene from the beginning. In response to 

30 selection of the "Stop-Continue" option 116 with 
the cursor 58 and graphical input 18 (point-and- 
click) or audio system 22 input, the computer will 
cause the replay of the audio-visual work to be 
halted. A subsequent selection of the "Stop-Con- 

35 tinue" option 116 will cause playback to continue, 
commencing from the point at which it was pre- 
viously stopped. By selecting one of the queuing 
controls 120 with the cursor 58 and graphical input 
18 (point-and-click) or audio system 22 input, the 

40 user may move forward and backward from scene 
to scene in the audio-visual work. In response to 
the selection of the "Scene Menu" option 122 with 
the graphical input 18 (point-and-click) or audio 
system 22 input, the system 10 will return to the 

45 Scene Menu 78. Furthermore, in response to the 
selection of the "Return" option 124 with the cursor 
58 and graphical input 18 (point-and-click) or audio 
system 22 input, the system 10 will return to the 
Main Menu 44. 

so While the audio-visual work is being displayed 

in window 112 using screen 110, the system 10 
makes several interactive feature 125 options avail- 
able for user access. The screen 110 further in- 
cludes windows 126 and 128. By selecting window 

55 126 with the cursor 58 and graphical input 18 
(point-and-click) or audio system 22 input, the 
computer 12 retrieves from floppy disc 32 and 
synchronously displays in window 126 a transcrip- 
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tion 132 of the audio portion of the audio-visual 
work for user consideration. Similarly, by selecting 
window 128 with the cursor 58 and graphical input 
18 (point-and-click) or audio system 22 input, the 
computer 12 retrieves from floppy disc 32 and 
synchronously displays in window 128 a translation 
134 of the audio portion of the audio-visual work for 
user consideration in the language selected by the 
user with the "Language" option 80 of the Scene 
Menu 78. 

The screen 110 further includes a window 130. 
With the display of the audio transcription and/or 
translation in windows 126 and/or 128, another op- 
tion is made available as the computer 12 will 
highlight or flag 136 certain items, words or 
phrases displayed in windows 126 and 128 to 
catch the attention of the user. Selection of this 
option is made by accessing the "Notes" option 
118 with the cursor 58 and graphical input 18 
(point-and-click) or audio system 22 input. By 
thereafter selecting one of the highlighted or 
flagged items, words or phrases with the cursor 58 
and graphical input 18 (point-and-click) or audio 
system 22 input, the computer 12 pauses playback 
of the audio-visual work, retrieves information con- 
cerning the selected item, word or phrase from 
floppy disc 32, and displays the information for 
user consideration in window 130. The type of 
highlighting or flagging used on an item, word or 
phrase (for example, the choice of a particular 
color, font, or icon) differs with respect to the type 
of information that will be retrieved if the item, word 
or phrase is selected. For example, the displayed 
information may comprise a definition of the se- 
lected word or phrase retrieved from a dictionary 
whose contents are selectively stored on the floppy 
disc 32 or mass storage 28. In addition, the dis- 
played information may comprise a grammatical 
note concerning the selected word or phrase and 
its use in the displayed sentence. Furthermore, the 
displayed information may comprise historical, cul- 
tural or geographic information (taken from, for 
example, encyclopedic references or a map) con- 
cerning the selected item, word or phrase or as- 
pects of the selected scene currently being viewed 
by the user. 

Other interactive feature 125 options for assist- 
ing with student learning are also made available 
for user selection while the audio-visual work is 
being displayed using screen 112. For example, 
the user may select a "fill-in-the-blanks" option 138 
through user entry of an appropriate command with 
the textual input 16 or audio system 22 input. 
Alternatively, the fill-in-the-blanks option 138 is 
automatically executed by the computer 12 at cer- 
tain pre-programmed points in the playback of the 
audio-visual work. With execution of the fill-in-the- 
blanks option 138, the computer 12 provides a 



transcription 132 of a line of dialog or a short 
portion of a conversation in window 126 with ran- 
domly or purposefully selected words left blank. 
Playback of the audio-visual work by the computer 

5 12 is then paused to allow the user to consider the 
previously played portion of the work and choose 
words in accordance with the user's recollection of 
the dialog to fill in the provided blank spaces. In 
addition, the chosen words may be input into the 

io computer 12 via the textual input 16 or audio 
system 22 input. The computer 12 will then display 
the correct answers for user consideration and 
comparison to the chosen words. In the event of 
user input of the words, the computer 12 will pro- 

/5 cess the input (in accordance with the speech 
recognition functionality of the processor 24, if 
needed), and compare the input words with the 
correct words, indicating on the displayed tran- 
scription whether the user input was correct. This 

20 fill-in-the-blanks feature 138 facilitates the enhance- 
ment of user transcription and oral comprehension 
skills by making the user carefully listen to the 
audio portion of the audio-visual work. The fill-in- 
the-blanks option 138 is also available, if selected 

25 with the appropriate command, in connection with a 
display of a translation 134 of the dialog in window 
128. In this instance, the feature facilitates en- 
hancement of user translation and oral comprehen- 
sion skills. 

30 As the user advances in skill level, still other 

interactive feature 125 options for assisting with 
student learning are available for user selection 
while the audio-visual work is being displayed us- 
ing screen 112. For example, the user may select a 

35 "role play" option 140 by inputting an appropriate 
command using textual input 16 or audio system 
22 input. Alternatively, the role play option 140 is 
automatically executed by the computer 12 at cer- 
tain system or teacher pre-programmed points in 

40 the playback of the audio-visual work. During ex- 
ecution of the "role play" option 140, the user 
becomes an active participant in the drama being 
displayed by the audio-visual work. The computer 
12 will pause playback of the audio-visual work 

45 after a line of dialog or a certain scene and allow 
the user to supply, in context with the plot of the 
work from the point of view of a selected character, 
an appropriate response to a spoken line of dialog. 
Such a user formulated response may be input into 

so the computer 12 via the textual input 16 or audio 
system 22 input for display (after recognition if an 
audible input) in window 130. To facilitate effective 
foreign language learning, the user responses 
made during role play may be formulated and/or 

55 input in either or both the foreign language to be 
learned and/or the native (or other) language se- 
lected by the user. By selecting the "Continue" 
option 116, the playback of the audio-visual work 
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continues and next line of dialog is spoken, with 
the transcription and translation displayed in win- 
dows 126 and 128, respectively, for user compari- 
son to the formulated and/or entered (shown in 
window 130) user role play response. This feature 
tests not only user oral and written comprehension 
and understanding of the language, but also the 
user's ability to effectively engage in a conversa- 
tion based on a viewed context. 

Furthermore, as the student becomes more 
proficient at the selected foreign language, tran- 
scription and translation input options 142 and 144, 
respectively, may be selected. In either option, the 
computer 12 will cause the playback of a line of 
dialog or short conversation from the audio-visual 
work. In the transcription input .option 142, the 
student will formulate a transcription of the line of 
dialog or short conversation and preferably use the 
textual input 16 to enter the transcription for display 
in window 130. Alternatively, the user may input 
the transcription using the audio system 22 input 
and the speech recognition functionality of the pro- 
cessor 24. Similarly, in the translation input option 
144, the student will formulate and preferably enter 
a translation of the line of dialog or short conversa- 
tion. Following formulation and/or entry, the user 
may select window 126 or 128, as appropriate, with 
the cursor 58 and graphical input 18 (point-and- 
click) or audio system 22 input, and the computer 
12 will display the correct transcription or transla- 
tion (or both) for user comparison against their 
formulated transcription or translation, and/or com- 
puter comparison against their entered transcription 
or translation. Again, such a feature tests user oral 
and written comprehension and understanding of 
the foreign language. 

To assist the student in practicing their foreign 
language pronunciation, yet another interactive fea- 
ture 125 learning option is made available for user 
selection during display of the audio-visual work. 
When the pronunciation practice option 146 is se- 
lected by the user through the entry of an appro- 
priate command with the textual input 16 or audio 
system 22 input, the computer 12 will cause the 
playback of a short portion of the audio-visual work 
(perhaps just one or two words or a phrase) for the 
user to listen to and watch. The system 10 then 
pauses and allows the user to audibly repeat the 
short portion, with the spoken repetition received 
by the microphone of the audio system 22 and 
recorded by the computer 12. The user spoken 
repetition and actual audio from the audio-visual 
work are then replayed by the computer 12 for the 
user to hear so that the user may make an audible 
comparison and further practice proper pronunci- 
ation. In addition, or in the alternative, the audible 
input of the user may be processed for recognition 
by the speech recognition functionality of the pro- 



cessor 24. To further assist the user in the use of 
the pronunciation practice option 146 (or in the 
transcription and translation options mentioned 
above, if necessary), the rate of speech for the 

5 audio portion of the audio-visual work may be 
slowed down by the computer 12 upon user com- 
mand while introducing only a minimal, if any, 
distortion in the sound. This allows the user to 
carefully and slowly listen to individual words in 

w fast-paced conversations, such as in arguments, or 
in instances where multiple characters are talking 
at the same time. 

Another interactive feature 125 option made 
available to the user in connection with the display 

75 of the dialog transcription and/or translation is the 
selection of a displayed word or phrase for storage 
and subsequent review. Each selected word or 
phrase (along with a definition and use context) is 
stored in a study file 148 maintained by the user. 

20 The computer 12 then makes several word pro- 
cessing options available to the user for manipulat- 
ing the study file to facilitate user learning of the 
selected words and phrases. For example, the list 
may be alphabetized, randomized, edited, etc. Fur- 

25 thermore, all or part of the study file may be output 
by the computer 12 in a hard copy for further user 
review away from the system 10. 

In response to the selection of the "Quit" op- 
tion 56 from the main menu 44 with the cursor 58 

30 and graphical input 18 (point-and-click) or audio 
system 22 input, the computer 12 will end the 
current learning session, close any system pro- 
gram files, close all user files, and return to the 
computer operating system for entry of a com- 

35 mand. 

Although an embodiment of the present inven- 
tion has been illustrated in the accompanying 
Drawings and described in the foregoing Detailed 
Description, it will be understood that the invention 
40 is not limited to the embodiment disclosed, but 
may be modified without departing from the scope 
of the invention as set forth and defined by the 
following claims. 

45 Claims 

1. A method for teaching a student with an inter- 
active audio-visual presentation system includ- 
ing a playback device for accessing a medium 
50 storing audio-visual work and a display device 

for displaying the audio-visual work, compris- 
ing the steps of: 

storing textual information associated with 
the audio-visual work; 
55 controlling the playback device to selec- 

tively display a portion of the stored audio- 
visual work for student viewing; 

retrieving and displaying, in conjunction 
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and 



with the display of the portion of the stored 
audio-visual work, an associated portion of the 
stored textual information; and 

pausing playback of the audio-visual work 
and display of the associated textual informa- 
tion to allow the student to respond to the 
displayed portion of the associated textual in- 
formation. 

2. The method as in Claim 1 further including the 
steps of: 

inputting the response from the student; 

comparing the input response with a cor- 
rect response. 

3. The method as in claim 1 further including the 
steps of: 

audibly inputting the response by the stu- 
dent; 

speech recognizing the audibly input re- 
sponse; and 

comparing the recognized audibly input re- 
sponse with a correct response. 

4. The method as in Claim 1 wherein the step of 
storing comprises the step of storing dialog for 
the audio-visual work, wherein the step of re- 
trieving and displaying comprises the step of 
displaying a line of dialog from the audio-visual 
work, and wherein the step of pausing com- 
prises the step of pausing to allow the student 
to respond with a guess as to a responsive line 
of dialog next spoken in the audio-visual work, 
the method further including the steps of: 

continuing playback of the audio-visual 
work; 

retrieving and displaying an actual respon- 
sive line of dialog next spoken in the audio- 
visual work; and 

comparing the guess of the responsive line 
of dialog by the student with the displayed 
actual responsive line of dialog next spoken in 
the audio-visual work. 

5. The method as in Claim 1 wherein the step of 
storing comprises the step of storing dialog for 
the audio-visual work, wherein the step of re- 
trieving and displaying comprises the step of 
displaying a line of dialog from the audio-visual 
work with selected words therein left blank, 
and wherein the step of pausing comprises the 
step of pausing to allow the student to respond 
with a guess as to the selected words left 
blank in the displayed line of dialog, the meth- 
od further including the steps of: 

retrieving and displaying a completed line 
of dialog; and 



comparing each student guess of the se- 
lected words with the displayed completed line 
of dialog. 

5 6. The method as in Claim 1 wherein the step of 
storing comprises the step of storing a plurality 
of comprehension testing questions and cor- 
rect answers concerning the audio-visual work, 
and wherein the step of retrieving and display- 

w ing comprises the step of displaying one or 

more comprehension testing questions con- 
cerning the displayed portion of the audio- 
visual work, and wherein the step of pausing 
comprises the step of pausing to allow the 

;5 student to respond with a guess as to an 

answer to each displayed comprehension test- 
ing question, the method further comprising 
the steps of: 

retrieving and displaying the correct an- 

20 swers to each displayed comprehension test- 

ing question; and 

comparing the student guess of the an- 
swers with the correct answers to each ►dis- 
played comprehension testing question. 

25 

7. The method as in Claim 1 wherein the step of 
storing includes the step of storing a list of 
scenes in the audio-visual work along with 
scene synopses, and wherein the step of re- 

30 trieving and displaying includes the step of 

displaying the list of scenes for student review, 
the method further including the steps of: 

inputting a selection by the student of a 
particular scene from the list of scenes; 

35 displaying the synopsis of the selected 

scene; 

inputting a selection by the student of a 
particular scene from the list of scenes; and 
displaying the selected scene from the 
40 audio-visual work for student viewing. 

8. The method as in Claim 1 wherein the step of 
storing includes the step of storing a transcrip- 
tion of the dialog spoken on the audio-visual 

45 work, and wherein the step of retrieving and 

displaying includes the step of displaying the 
transcription of the dialog in conjunction with 
the display of the audio-visual work, the meth- 
od further including the steps of: 

so flagging portions of the displayed tran- 

scription of the dialog; 

inputting a selection by the student of a 
flagged portion in the displayed transcription of 
the dialog; and 

55 displaying information concerning the 

flagged portion. 
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9. The method as in Claim 8 wherein the step of 
displaying information concerning the flagged 
portion comprises the step of displaying a defi- 
nition for a selected word in the transcription. 

10. The method as in Claim 9 wherein the step of 
displaying information concerning the flagged 
portion comprises the step of displaying gram- 
matical information for a selected word in the 
transcription. 

11. The method as in Claim 8 wherein the step of 
displaying information concerning the flagged 
portion comprises the step of displaying cul- 
tural information for the audio-visual work. 

12. The method as in Claim 8 wherein the step of 
displaying information concerning the flagged 
portion comprises the step of displaying his- 
torical information for the audio-visual work. 

13. The method as in Claim 8 wherein the step of 
displaying information concerning the flagged 
portion comprises the step of displaying geo- 
graphic information for the audio-visual work. 

14. A method for teaching a student with an inter- 
active audio-visual presentation system includ- 
ing a playback device for accessing a medium 
storing an audio-visual work and a display de- 
vice for displaying the audio-visual work, com- 
prising the steps of: 

storing a dialog spoken on the audio-visual 
work; 

controlling the playback device to selec- 
tively display a portion of the stored audio- 
visual work for student viewing; 

pausing playback of the audio-visual work; 

inputting a guess by the student as to the 
dialog spoken with the displayed portion of the 
audio-visual work; 

retrieving and displaying the stored dialog 
spoken with the displayed portion of the audio- 
visual work; and 

comparing the student guess as to the 
spoken dialog with the stored dialog actually 
spoken with the displayed portion of the audio- 
visual work. 

15. The method as in Claim 14 wherein the step of 
storing comprises the step of storing a tran- 
scription of each spoken line of dialog, and 
wherein the step of inputting comprises the 
step of inputting a guess by the student as to 
the transcription of the dialog spoken with the 
displayed portion. 



16. The method as in Claim 15 wherein the step of 
retrieving and displaying includes the step of 
displaying the transcription of each spoken line 
of dialog in conjunction with a re-display of the 

5 displayed portion of the stored audio-visual 

work. 

17. The method as in Claim 14 wherein the audio- 
visual work is a foreign language work, and 

70 wherein the step of storing comprises the step 

of storing a translation of each spoken line of 
dialog, and wherein the step of inputting com- 
prises the step of inputting a guess by the 
student as to the translation of the dialog spok- 

75 en in the displayed portion. 

18. The method as in Claim 17 wherein the step of 
retrieving and displaying includes the step of 
displaying the translation in conjunction with a 

20 re-display of the previously displayed portion 

of the stored foreign language audio-visual 
work. 

19. The method as in Claim 14 wherein the step of 
25 inputting comprises the step of audibly input- 
ting the guess by the student, further including 
the step of speech recognizing the audibly 
input guess for comparison to the dialog ac- 
tually spoken. 

30 

20. A system for interactive student teaching, com- 
prising: 

a playback device for playing an audio- 
visual work; 
35 a display device; and 

a computer connected to the playback de- 
vice and display device including: 

means for storing textual information asso- 
ciated with the audio-visual work; 
40 means for controlling the playback device 

to selectively display on the display device for 
student viewing a portion of the stored audio- 
visual work; 

means for retrieving an associated portion 
45 of the stored textual information for display on 

the display device in conjunction with the por- 
tion of the stored audio-visual work; and 

means for pausing playback of the audio- 
visual work and display of the associated tex- 
50 tual information to allow the student to respond 

to the displayed portion of the associated tex- 
tual information. 

21. The system as in Claim 20 wherein the com- 
55 puter further includes: 

means for allowing the student to input the 
response to the displayed portion of the asso- 
ciated textual information; and 



10 



19 



EP 0 665 523 A2 



20 



means for comparing the input response to 
a correct response. 

22. The system as in Claim 20 wherein the com- 
puter further includes: 

means for audibly inputting by the student 
the response; 

means for speech recognizing the audibly 
input response; and 

means for comparing the recognized au- 
dibly input response to a correct response. 

23. The system as in Claim 20 wherein the asso- 
ciated textual information comprises dialog for 
the audio-visual work, wherein the displayed 
portion of the associated textual information 
comprises a line of dialog from the audio- 
visual work, and wherein the response by the 
student comprises a guess by the student as 
to the next line of dialog spoken in the audio- 
visual work, the computer further including: 

means for continuing playback of the 
audio-visual work; 

means for displaying an actual responsive 
line of dialog next spoken in the audio-visual 
work; and 

means for comparing the actual responsive 
line of dialog with the student guess as to the 
next line of dialog. 

24. The system as in Claim 20 wherein the asso- 
ciated textual information comprises dialog for 
the audio-visual work, wherein the displayed 
portion of the associated textual information 
comprises a line of dialog from the audio- 
visual work with selected words therein left 
blank, and wherein the response by the stu- 
dent comprises a guess by the student as to 
the words left blank in the displayed line of 
dialog, the computer further including: 

means for retrieving for display a com- 
pleted line of dialog; and 

means for comparing the completed line of 
dialog with the student guess as to the se- 
lected blank words in the line of dialog. 

25. The system as in Claim 20 wherein the textual 
information comprises a plurality of compre- 
hension testing questions concerning the 
audio-visual work along with correct answers 
thereto, and wherein the means for retrieving 
and displaying includes means for displaying 
one or more comprehension testing questions 
concerning the displayed portion of the audio- 
visual work, and wherein the student response 
comprises a guess as to an answer to each 
displayed comprehension testing question, the 
computer further including: 



means for retrieving for display the correct 
answer to each displayed comprehension test- 
ing question; and 

means for comparing the correct answers 
5 to each formulated student guess. 

26. A system for interactive student teaching, com- 
prising: 

a playback device for playing an audio- 
w visual work; 

a display device; and 

a computer connected to the playback de- 
vice and the display device including: 

means for storing dialog spoken on the 
75 audio-visual work; 

means for controlling the playback device 
to selectively display for student viewing a 
portion of the stored audio-visual work; 

means for pausing playback of the audio- 
20 visual work by the playback device to allow the 

student to input a guess as to the dialog spok- 
en on the displayed portion of the audio-visual 
work; and 

means for retrieving the stored dialog 
25 spoken on the displayed portion of the audio- 

visual work for display and comparison to the 
input student guess as to the dialog. 

27. The system as in Claim 26 wherein the means 
30 for storing comprises means for storing a tran- 
scription of the spoken line of dialog, the input 
student guess comprising a guess as to the 
transcription of the dialog spoken in the dis- 
played portion. 

35 

28. The system as in Claim 26 wherein the audio- 
visual work is a foreign language work, and 
wherein the means for storing comprises 
means for storing a translation of the spoken 

40 line of dialog, the input student guess compris- 

ing a guess as to the translation of the dialog 
spoken in the displayed portion. 
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